On the Dirichlet Prior and Bayesian Regularization

Motivation & Previous Work: A common objective in learning a model from data is to recover its network structure, while the model parameters are of minor interest. For example, we may wish to recover regulatory networks from high-throughput data sources. Regularization is essential when learning from finite data sets. It provides not only smoother estimates of the model parameters compared to maximum likelihood but also guides the selection of model structures. In the Bayesian approach, regularization is achieved by specifying a prior distribution over the parameters and subsequently averaging over the posterior distribution. In domains comprising discrete variables with a multinomial distribution, the Dirichlet distribution is the most commonly used prior over the parameters. This is because of the following two reasons: first, the Dirichlet distribution is the conjugate prior to the multinomial distribution and hence permits analytical calculations, and second, the Dirichlet prior is intimately tied to the desirable likelihood-equivalence property of network structures [1, 3]. The so-called equivalent sample size measures the strength of the prior belief. In [3], it was pointed out that a very strong prior belief can degrade predictive accuracy of the learned model due to severe regularization of the parameter estimates. In contrast to that, the dependence of the learned network structure on the prior strength has not received much attention in the literature, despite its relevance for the recovery of the ”true” network structure underlying the data.